Displacement measurement device and displacement measurement method

ABSTRACT

A displacement measurement device includes: an obtainer that obtains a first image which contains a subject and a second image which contains the subject; a generator that generates M template images which contain the subject and which have noise from the first image and generates M target images which contain the subject and which have noise from the second image, M being an integer of 2 or higher; a hypothetical displacement calculator that calculates M hypothetical displacements of the subject from the M template images and the M target images; and a displacement calculator that calculates a displacement of the subject by performing statistical processing on the M hypothetical displacements.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a U.S. continuation application of PCT InternationalPatent Application Number PCT/JP2019/029116 filed on Jul. 25, 2019,claiming the benefit of priority of Japanese Patent Application Number2018-161936 filed on Aug. 30, 2018 and Japanese Patent ApplicationNumber 2018-198570 filed on Oct. 22, 2018, the entire contents of whichare hereby incorporated by reference.

BACKGROUND 1. Technical Field

The present disclosure relates to a displacement measurement device anda displacement measurement method that measure displacement of a subjectto be measured.

2. Description of the Related Art

Techniques which use first and second images in which a subject to bemeasured has been captured to measure displacement of that subject havebeen known for some time (see, for example, Japanese Unexamined PatentApplication Publication No. 2006-254349).

When capturing an image of the subject to be measured, noise caused byfluctuations in the atmosphere of the space between an image capturingdevice and the subject due to heat, wind, or the like, noise caused bywater droplets falling during rainfall, and the like may appear in thecaptured image. Additionally, the captured image may have shot noisefrom the outset.

According to the past technique, when measuring the displacement of asubject, the accuracy of the measurement will be lower if the image usedfor the measurement contains noise than if the image does not containnoise.

Accordingly, an object of the present disclosure is to provide adisplacement measurement device and a displacement measurement methodthat, when measuring the displacement of a subject from first and secondimages of the subject, can measure the displacement with a higher levelof accuracy than in the past.

SUMMARY

A displacement measurement device according to one aspect of the presentdisclosure includes: an obtainer that obtains a first image whichcontains a subject to be measured and a second image which contains thesubject; a generator that generates M template images which contain thesubject and which have noise from the first image and generates M targetimages which contain the subject and which have noise from the secondimage, M being an integer of 2 or higher; a hypothetical displacementcalculator that calculates M hypothetical displacements of the subjectfrom the M template images and the M target images; and a displacementcalculator that calculates a displacement of the subject by performingstatistical processing on the M hypothetical displacements.

A displacement measurement method according to one aspect of the presentdisclosure includes: obtaining a first image which contains a subject tobe measured and a second image which contains the subject; generating Mtemplate images which contain the subject and which have noise from thefirst image and generating M target images which contain the subject andwhich have noise from the second image, M being an integer of 2 orhigher; calculating M hypothetical displacements of the subject from theM template images and the M target images; and calculating adisplacement of the subject by performing statistical processing on theM hypothetical displacements.

With the displacement measurement device and displacement measurementmethod according to one aspect of the present disclosure, when measuringthe displacement of a subject from first and second images of thesubject, the displacement can be measured with a higher level ofaccuracy than in the past.

BRIEF DESCRIPTION OF DRAWINGS

These and other objects, advantages and features of the disclosure willbecome apparent from the following description thereof taken inconjunction with the accompanying drawings that illustrate a specificembodiment of the present disclosure.

FIG. 1A is a conceptual diagram schematically illustrating an example ofa relationship between true displacement and displacement with noise;

FIG. 1B is a conceptual diagram schematically illustrating thereproduction of true displacement from a plurality of template imagescontaining noise and a plurality of target images containing noise;

FIG. 2 is an exterior view illustrating an example of the configurationof a displacement measurement system according to Embodiment 1;

FIG. 3 is a block diagram illustrating the configuration of adisplacement measurement device according to Embodiment 1;

FIG. 4 is a schematic diagram illustrating an example of learning by afirst learning model according to Embodiment 1;

FIG. 5 is a schematic diagram illustrating an example of a firstgenerator according to Embodiment 1 generating M template images and Mtarget images;

FIG. 6 is a schematic diagram illustrating an example of a hypotheticaldisplacement calculator according to Embodiment 1 calculating Mhypothetical displacements;

FIG. 7 is a schematic diagram illustrating an example of a displacementcalculator according to Embodiment 1 calculating a displacement;

FIG. 8 is a flowchart illustrating first learning processing accordingto Embodiment 1;

FIG. 9 is a flowchart illustrating first displacement measurementprocessing according to Embodiment 1;

FIG. 10 is a block diagram illustrating the configuration of adisplacement measurement device according to Embodiment 2;

FIG. 11 is a schematic diagram illustrating an example of theconfiguration of a first machine learning model according to Embodiment2;

FIG. 12 is a schematic diagram illustrating an example of learning bythe first machine learning model according to Embodiment 2;

FIG. 13 is a flowchart illustrating second learning processing accordingto Embodiment 2;

FIG. 14 is a block diagram illustrating the configuration of adisplacement measurement device according to Embodiment 3;

FIG. 15 is a flowchart illustrating third learning processing accordingto Embodiment 3;

FIG. 16 is a block diagram illustrating the configuration of adisplacement measurement device according to Embodiment 4;

FIG. 17 is a block diagram illustrating the configuration of a firstgenerator according to Embodiment 4;

FIG. 18 is a block diagram illustrating the configuration of adisplacement measurement device according to Embodiment 5;

FIG. 19A is a schematic diagram illustrating an example of arelationship between a probability density of light incident on a pixeland a pixel value read out from that pixel;

FIG. 19B is a schematic diagram illustrating an example of arelationship between a probability density of light incident on a pixeland a pixel value read out from that pixel; and

FIG. 20 is a flowchart illustrating third displacement measurementprocessing according to Embodiment 5.

DETAILED DESCRIPTION OF THE EMBODIMENTS Background Leading toRealization of One Embodiment of the Present Disclosure

When using first and second images in which a subject to be measured iscaptured to measure displacement of that subject, the displacementobtained as a result of the measurement may differ between a situationwhere there is noise in the images and a situation where there is nonoise in the images. Here, noise caused by fluctuations in theatmosphere of the space between an image capturing device and thesubject due to heat, wind, or the like, noise caused by water dropletsfalling during rainfall, and the like are conceivable as noise which mayappear in the image. Additionally, the captured image may have shotnoise from the outset.

FIG. 1A is a conceptual diagram schematically illustrating an example ofa relationship between displacement of the subject obtained as a resultof measurement from images which do not contain noise (also called “truedisplacement” hereinafter) and displacement of the subject obtained as aresult of measurement from images which do contain noise (also called“displacement with noise” hereinafter).

As illustrated in FIG. 1A, a displacement with noise obtained from asingle template image captured with noise and a single target imagecaptured with noise may contain error with respect to the truedisplacement.

The inventors believed that the single images captured with noise aremerely images captured at an instant where a single given pattern ofnoise, out of a variety of possible noise patterns, is present. Theinventors furthermore thought that if a plurality of template imageshaving noise in a variety of possible noise patterns could be reproducedfrom the single template image captured with noise, and a plurality oftarget images having noise in a variety of possible noise patterns couldbe reproduced from the single target image captured with noise, then aplurality of displacements with noise corresponding to the variety ofpossible noise patterns could be reproduced as well. The inventorspostulated that the true displacement could be restored by performingstatistical processing on the reproduced plurality of displacements withnoise.

FIG. 1B is a conceptual diagram schematically illustrating thereproduction of a true displacement from a plurality of template imagescontaining noise and a plurality of target images containing noise.

As illustrated in FIG. 1B, when a plurality of displacements with noiserepresent a statistical distribution (e.g., a Gaussian distribution)with respect to the true displacement, the true displacement can bereproduced by performing statistical processing on the plurality ofdisplacements with noise.

The inventors arrived at the displacement measurement device anddisplacement measurement method described below based on the foregoingideas.

A displacement measurement device according to one aspect of the presentdisclosure includes: an obtainer that obtains a first image whichcontains a subject to be measured and a second image which contains thesubject; a generator that generates M template images which contain thesubject and which have noise from the first image and generates M targetimages which contain the subject and which have noise from the secondimage, M being an integer of 2 or higher; a hypothetical displacementcalculator that calculates M hypothetical displacements of the subjectfrom the M template images and the M target images; and a displacementcalculator that calculates a displacement of the subject by performingstatistical processing on the M hypothetical displacements.

The above-described displacement measurement device generates M templateimages and M target images from one first image and one second image,respectively. Then, the device calculates M hypothetical displacementsfrom the M template images and the M target images, and calculates thedisplacement of the subject by performing statistical processing on thecalculated hypothetical displacements. Thus according to theabove-described displacement measurement device, the displacement can becalculated more accurately than by a past type of displacementmeasurement device which calculates the displacement directly from onefirst image and one second image, without performing statisticalprocessing.

Additionally, the generator may generate the M template images by addingnoise to the first image, and generate the M target images by addingnoise to the second image.

Through this, a drop in the accuracy of the measurement of thedisplacement of the subject, caused by shot noise, can be suppressed.

Additionally, based on a pixel value of a pixel to which noise is to beadded, the generator may add the noise to the pixel so that as the pixelvalue decreases, a greater amount of noise is added.

Through this, a drop in the accuracy of the measurement of thedisplacement of the subject, caused by shot noise, can be moreeffectively suppressed.

Additionally, the generator may generate the M template images byadding, to the first image, image capturing device-originating noiseproduced by an image capturing device that captured the first image, andgenerate the M target images by adding, to the second image, imagecapturing device-originating noise produced by an image capturing devicethat captured the second image.

Through this, a drop in the accuracy of the measurement of thedisplacement of the subject, caused by image capturingdevice-originating noise, can be suppressed.

Additionally, based on pixel values of pixels to which the imagecapturing device-originating noise is to be added, the generator addsthe image capturing device-originating noise to the pixels so that asthe pixel value decreases, a greater amount of noise originating fromthe image capturing device is added.

Through this, a drop in the accuracy of the measurement of thedisplacement of the subject, caused by image capturingdevice-originating noise, can be more effectively suppressed.

Additionally, the image capturing device-originating noise may be darkcurrent noise of the image capturing device that captured the image.

Through this, a drop in the accuracy of the measurement of thedisplacement of the subject, caused by dark current noise of the imagecapturing device, can be suppressed.

Additionally, the image capturing device-originating noise may bethermal noise of the image capturing device that captured the image.

Through this, a drop in the accuracy of the measurement of thedisplacement of the subject, caused by thermal noise of the imagecapturing device, can be suppressed.

A displacement measurement method according to one aspect of the presentdisclosure includes: obtaining a first image which contains a subject tobe measured and a second image which contains the subject; generating Mtemplate images which contain the subject and which have noise from thefirst image and generating M target images which contain the subject andwhich have noise from the second image, M being an integer of 2 orhigher; calculating M hypothetical displacements of the subject from theM template images and the M target images; and calculating adisplacement of the subject by performing statistical processing on theM hypothetical displacements.

The above-described displacement measurement method generates M templateimages and M target images from one first image and one second image,respectively. Then, M hypothetical displacements are calculated from theM template images and the M target images, and the displacement of thesubject is calculated by performing statistical processing on thecalculated hypothetical displacements. Thus according to theabove-described displacement measurement method, the displacement can becalculated more accurately than by a past type of displacementmeasurement method which calculates the displacement directly from onefirst image and one second image, without performing statisticalprocessing.

Specific examples of the displacement measurement device anddisplacement measurement method according to aspects of the presentdisclosure will be described hereinafter with reference to the drawings.Each of the following embodiments describes a specific example of thepresent disclosure. As such, the numerical values, shapes, constituentelements, arrangements and connection states of constituent elements,steps, orders of steps, and the like in the following embodiments aremerely examples, and are not intended to limit the present disclosure.Additionally, of the constituent elements in the following embodiments,constituent elements not denoted in the independent claims areconsidered to be optional constituent elements. Additionally, thedrawings are schematic diagrams, and are not necessarily exactillustrations.

Note that these comprehensive or specific aspects of the presentdisclosure may be realized by a system, a method, an integrated circuit,a computer program, or a computer-readable recording medium such as aCD-ROM, or may be implemented by any desired combination of systems,devices, methods, integrated circuits, computer programs, and recordingmedia.

Embodiment 1

1-1. Overview of Displacement Measurement System

A displacement measurement system that captures a plurality of images ofa subject to be measured and calculates a displacement of the subjectfrom the plurality of captured images, and a displacement measurementdevice included in the displacement measurement system, will bedescribed here.

FIG. 2 is an exterior view illustrating an example of the configurationof displacement measurement system 1 according to Embodiment 1.

As illustrated in FIG. 2, displacement measurement system 1 isconfigured including image capturing device 200 and displacementmeasurement device 100.

Image capturing device 200 captures an image of subject 300, which is asubject to be measured. Image capturing device 200 captures, forexample, a plurality of images of subject 300 over time, from a fixedangle of view.

For example, when displacement measurement device 100 is to train firstmachine learning model 10 (described later), image capturing device 200captures a plurality of images of subject 300 at a time when there is nodisplacement in subject 300, i.e., when a load acting on subject 300 isnot changing. For example, if subject 300 is a bridge, image capturingdevice 200 captures a plurality of images at a time when no vehicles aretraveling on the bridge.

Additionally, for example, when displacement measurement device 100 isto measure a displacement of subject 300, image capturing device 200captures a plurality of images of subject 300 during a period in whichthere is displacement in subject 300 and/or a period in which there isno displacement, i.e., during a period in which a load acting on subject300 is changing and/or a period in which the load is not changing. Inother words, image capturing device 200 captures a plurality of imagesof subject 300 during a displacement measurement period of displacementmeasurement device 100. For example, if subject 300 is a bridge, imagecapturing device 200 captures a plurality of images during a period inwhich no vehicles are traveling on the bridge and/or a period in whichvehicles are traveling on the bridge.

Image capturing device 200 includes a communication function, andcommunicates with an external device. The external device includesdisplacement measurement device 100. Image capturing device 200 maycommunicate with the external device through wired communication, or maycommunicate with the external device through wireless communication, forexample.

Image capturing device 200 is implemented by a digital video camera or adigital still camera including an image sensor, for example.

An image captured by image capturing device 200 may have noise. Here,noise caused by fluctuations in the atmosphere of the space between animage capturing device and the subject due to heat, wind, or the like,noise caused by water droplets falling during rainfall, and the like areconceivable as noise which may appear in the image.

Displacement measurement device 100 calculates a displacement of subject300 from a plurality of images, captured by image capturing device 200,which contain subject 300.

Displacement measurement device 100 includes a communication function,and communicates with an external device. The external device includesimage capturing device 200. Displacement measurement device 100 maycommunicate with the external device through wired communication, or maycommunicate with the external device through wireless communication, forexample.

Displacement measurement device 100 is implemented in, for example, acomputer device including a processor and memory, by the processorexecuting a program stored in the memory.

Displacement measurement device 100 will be described in detail nextwith reference to the drawings.

1-2. Configuration of Displacement Measurement Device 100

FIG. 3 is a block diagram illustrating the configuration of displacementmeasurement device 100.

As illustrated in FIG. 3, displacement measurement device 100 isconfigured including first machine learning model 10, first obtainer 20,first generator 30, hypothetical displacement calculator 40,displacement calculator 50, second obtainer 60, and first trainer 70.

First machine learning model 10 is a machine learning model trained togenerate at least one image which contains subject 300 and has noise,from one image which contains subject 300 and has noise. First machinelearning model 10 is trained by first trainer 70. The training of firstmachine learning model 10 by first trainer 70 will be described later.

Second obtainer 60 obtains, from image capturing device 200, N (where Nis an integer of 2 or higher) images which contain subject 300 and havenoise. The N images obtained by second obtainer 60 are used in thetraining of first machine learning model 10 (described later). Theimages used in the training of first machine learning model 10(described later) are, for example, images captured when there is nodisplacement in subject 300. Accordingly, second obtainer 60 obtains Nimages captured when there is no displacement in subject 300.

With each of the N images obtained by second obtainer 60, first trainer70 trains first machine learning model 10 to generate at least one imagewhich contains subject 300 and has noise from one image which containssubject 300 and has noise, by using each of the N images as an input andusing at least one of N−1 other images as a correct answer.

FIG. 4 is a schematic diagram illustrating an example of first machinelearning model 10 being trained by first trainer 70.

A specific example of the training of first machine learning model 10 byfirst trainer 70 will be described next with reference to FIG. 4. Inthis example, image capturing device 200 captures a moving image ofsubject 300 when there is no displacement in subject 300, i.e., when aload acting on subject 300 is not changing, and second obtainer 60obtains N frames of the moving image captured by image capturing device200 as the N images which contain subject 300 and have noise.

As illustrated in FIG. 4, first trainer 70 sets, for each of the Nimages, at least one measurement point in a pixel region where subject300 appears. Here, each of the at least one measurement points is alocal region constituted by a plurality of pixels adjacent to eachother, and is a pixel region in a position that is the same throughoutthe N images. In the example illustrated in FIG. 4, first trainer 70sets measurement point 301 and measurement point 302 in each of the Nimages.

Then, first trainer 70 trains first machine learning model 10 with eachof the N images to generate at least one image which contains subject300 and has noise from one image which contains subject 300 and hasnoise, by using each of the N images as an input and using at least oneof the other N−1 images as a correct answer, and by performing thetraining on pixels included in each of the measurement points which havebeen set (measurement point 301 and measurement point 302). Here, firsttrainer 70 may train first machine learning model 10 on a measurementpoint-by-measurement point basis, or may train first machine learningmodel 10 using all measurement points simultaneously, for example.

The following will describe first trainer 70 as training first machinelearning model 10 to generate M (where M is an integer of 2 or higher)images which contain subject 300 and have noise. However, as anotherexample, first trainer 70 may train first machine learning model 10 togenerate one image which contains subject 300 and has noise from oneimage which contains subject 300 and has noise, and first machinelearning model 10 may generate M images which contain subject 300 andhave noise from one image which contains subject 300 and has noise bychanging a parameter of first machine learning model 10 M times whengenerating the image. As still another example, first trainer 70 maytrain first machine learning model 10 to generate one image whichcontains subject 300 and has noise from one image which contains subject300 and has noise, and first machine learning model 10 may generate Mimages which contain subject 300 and have noise from one image whichcontains subject 300 and has noise by using a Generative AdversarialNetwork (GAN) when generating the image.

Returning to FIG. 3, the descriptions of displacement measurement device100 will be continued.

First obtainer 20 obtains, from image capturing device 200, a firstimage which contains subject 300 and has noise and a second image whichcontains subject 300 and has noise. The first image and the second imageobtained by first obtainer 20 are used in the calculation of thedisplacement of subject 300 (described later). As such, first obtainer20 obtains the first image and the second image which have been capturedduring a period in which there is displacement in subject 300 and/or aperiod in which there is no displacement.

Using first machine learning model 10, first generator 30 generates Mtemplate images which contain subject 300 and have noise from the firstimage obtained by first obtainer 20, and generates M target images whichcontain subject 300 and have noise from the second image obtained byfirst obtainer 20.

FIG. 5 is a schematic diagram illustrating an example of first generator30 generating the M template images which have noise from the firstimage which has noise, and generating the M target images which havenoise from the second image which has noise, using first machinelearning model 10.

A specific example of the generation of the M template images from thefirst image and the generation of the M target images from the secondimage, performed by first generator 30 using first machine learningmodel 10, will be described next with reference to FIG. 5. In thisexample, image capturing device 200 captures a moving image of subject300 during a period in which there is displacement in subject 300 and/ora period in which there is no displacement, i.e., during a period inwhich a load acting on subject 300 is changing and/or a period in whichthe load is not changing; and first obtainer 20 obtains one frame of themoving image captured by image capturing device 200 as the first image,and another one frame as the second image.

As illustrated in FIG. 5, using first machine learning model 10, firstgenerator 30 generates the M template images which have noise from thefirst image which has noise, and generates the M target images whichhave noise from the second image which has noise, using the pixelsincluded in each of the measurement points set by first trainer 70(measurement point 301 and measurement point 302) as targets of thegeneration.

Returning to FIG. 3, the descriptions of displacement measurement device100 will be continued.

Hypothetical displacement calculator 40 calculates M displacements ofsubject 300 from the M template images which have noise and the M targetimages which have noise, generated by first generator 30. The Mdisplacements of subject 300 calculated by hypothetical displacementcalculator 40 from the M template images and the M target images will becalled “M hypothetical displacements of subject 300” hereinafter.

FIG. 6 is a schematic diagram illustrating an example of hypotheticaldisplacement calculator 40 calculating the M hypothetical displacementsfrom the M template images which have noise and the M target imageswhich have noise.

A specific example of the calculation of the M hypotheticaldisplacements from the M template images and the M target images byhypothetical displacement calculator 40 will be described next withreference to FIG. 6.

As illustrated in FIG. 6, hypothetical displacement calculator 40associates the M template images which have noise one-to-one with the Mtarget images which have noise to create M pairs of images. Then, foreach pair, hypothetical displacement calculator 40 calculates adisplacement of subject 300 in the template image and the target imageas the hypothetical displacement, performing the calculation on each ofthe measurement points set by first trainer 70 (measurement point 301and measurement point 302). Through this, hypothetical displacementcalculator 40 calculates the M hypothetical displacements of subject 300at each of the measurement points. In this example, hypotheticaldisplacement calculator 40 calculates x₁, x₂, . . . , x_(M) as thehypothetical displacements of subject 300 at measurement point 301, andcalculates y₁, y₂, . . . , y_(M) as the hypothetical displacements ofsubject 300 at measurement point 302.

Hypothetical displacement calculator 40 may calculate the displacementof subject 300 in the template image and the target image in each pairusing, for example, digital image correlation, or using, for example, asampling moiré method.

Hypothetical displacement calculator 40 may calculate a number of pixelsin the images as the hypothetical displacement, for example, or maycalculate a distance in real space as the hypothetical displacement, forexample.

Returning to FIG. 3, the descriptions of displacement measurement device100 will be continued.

Displacement calculator 50 calculates the displacement of subject 300 byperforming statistical processing on the M hypothetical displacementscalculated by hypothetical displacement calculator 40.

FIG. 7 is a schematic diagram illustrating an example of displacementcalculator 50 calculating the displacement of subject 300 by performingstatistical processing on the M hypothetical displacements.

A specific example of the calculation of the displacement of subject 300by displacement calculator 50 will be described hereinafter withreference to FIG. 7. Although the following will describe an example inwhich displacement calculator 50 calculates the hypotheticaldisplacement of subject 300 at measurement point 301, the same appliesto a case where the hypothetical displacement of subject 300 iscalculated at measurement point 302.

As illustrated in FIG. 7, displacement calculator 50 generates ahistogram for the M hypothetical displacements of subject 300 atmeasurement point 301 which have been calculated by hypotheticaldisplacement calculator 40. The hypothetical displacement with thehighest frequency is then specified from the histogram, and thespecified hypothetical displacement is calculated as the displacement ofsubject 300 at measurement point 301. However, as another example,displacement calculator 50 may calculate, for example, an average valueof the M hypothetical displacements of subject 300 at measurement point301 which have been calculated by hypothetical displacement calculator40, and use the calculated average value as the displacement of subject300 at measurement point 301; or, for example, may calculate a medianvalue and use the calculated median value as the displacement of subject300 at measurement point 301; or, for example, may calculate a trimmedmean and use the calculated trimmed mean as the displacement of subject300 at measurement point 301. As still another example, displacementcalculator 50 may perform fitting using an appropriate function (e.g., aGaussian function) on the histogram of the M hypothetical displacementsof subject 300 at measurement point 301 which have been calculated byhypothetical displacement calculator 40, and may then calculate thedisplacement of subject 300 at measurement point 301 on the basis of aresult of the fitting.

1-3. Operations of Displacement Measurement Device 100

Operations performed by displacement measurement device 100 configuredas described above will be described next.

Displacement measurement device 100 performs first learning processingand first displacement measurement processing. The first learningprocessing and the first displacement measurement processing performedby displacement measurement device 100 will be described in orderhereinafter.

The first learning processing is processing for training first machinelearning model 10 to generate at least one image which contains subject300 and has noise, from one image which contains subject 300 and hasnoise.

FIG. 8 is a flowchart illustrating the first learning processingperformed by displacement measurement device 100.

The first learning processing is started when a user using displacementmeasurement device 100 has performed an operation in displacementmeasurement device 100 for starting the first learning processing, when,for example, image capturing device 200 has captured a moving image ofsubject 300 in which there is no displacement in subject 300 and thereis noise in each of the frames constituting the captured moving image.

When the first learning processing is started, second obtainer 60obtains, from image capturing device 200, N frames of the framesconstituting the moving image captured of subject 300, as the N imageswhich contains subject 300 and have noise (step S100). In other words, Nimages captured when there is no displacement are obtained.

Once the N images having noise have been obtained, first trainer 70selects one image having noise from among the N unselected images havingnoise (step S110). Here, “unselected image” refers to an image, in aloop from the process of step S110 to the determination of “yes” in stepS130 (described later), which has not been selected in a past instanceof the process of step S110.

Once one image having noise has been selected, first trainer 70generates one instance of training data which takes the one selectedimage having noise as an input and the other N−1 images having noise ascorrect answers (step S120).

Once the one instance of training data has been generated, first trainer70 finds whether or not there is an unselected image among the N imageshaving noise (step S130).

If, in the process of step S130, there is an unselected image (stepS130: yes), displacement measurement device 100 moves the sequence tostep S110 again.

If, in the process of step S130, there is no unselected image (stepS130: no), first trainer 70 uses N instances of training data, obtainedby repeating the loop formed by the process of step S110 to thedetermination of “yes” in step S130 N times, to train first machinelearning model 10 to generate at least one image which contains subject300 and has noise from one image which contains subject 300 and hasnoise (step S140).

Once the process of step S140 ends, displacement measurement device 100ends the first learning processing.

The first displacement measurement processing is processing forcalculating the displacement of subject 300 from the first image, whichcontains subject 300 and has noise, and the second image, which containssubject 300 and has noise.

FIG. 9 is a flowchart illustrating the first displacement measurementprocessing performed by displacement measurement device 100.

The first displacement measurement processing is started when a userusing displacement measurement device 100 has performed an operation indisplacement measurement device 100 for starting the first displacementmeasurement processing, when, for example, image capturing device 200has captured a moving image of subject 300 during a period in whichthere is displacement in subject 300 and/or a period in which there isno displacement, and there is noise in each of the frames constitutingthe captured moving image.

When the first displacement measurement processing is started, firstobtainer 20 obtains, from image capturing device 200, one frame amongthe frames constituting the moving image captured of subject 300 as thefirst image which contains subject 300 and has noise, and obtainsanother one frame as the second image which contains subject 300 and hasnoise (step S200).

Once the first image having noise and the second image having noise havebeen obtained, first generator 30 generates the M template images whichhave noise from the first image which has noise, and generates the Mtarget images which have noise from the second image which has noise,using first machine learning model 10 (step S210).

Once the M template images having noise and the M target images havingnoise have been generated, hypothetical displacement calculator 40calculates the M hypothetical displacements from the M template imageshaving noise and the M target images having noise (step S220).

Once the M hypothetical displacements have been calculated, displacementcalculator 50 calculates the displacement of subject 300 by performingstatistical processing on the M hypothetical displacements (step S230).

Once the process of step S230 ends, displacement measurement device 100ends the first displacement measurement processing.

1-4. Effects

As described above, displacement measurement device 100 generates the Mtemplate images which have noise from the first image which has noise,and generates the M target images which have noise from the second imagewhich has noise. Then, displacement measurement device 100 calculatesthe M hypothetical displacements from the M template images having noiseand the M target images having noise, and calculates the displacement ofsubject 300 by performing statistical processing on the calculated Mhypothetical displacements. Accordingly, displacement measurement device100 can calculate the displacement more accurately than a past type ofdisplacement measurement device which calculates the displacement ofsubject 300 directly from a first image having noise and a second imagehaving noise, without performing statistical processing.

Embodiment 2

A displacement measurement device according to Embodiment 2, configuredby changing part of the configuration of displacement measurement device100 according to Embodiment 1, will be described next.

The displacement measurement device according to Embodiment 2 will bedescribed hereinafter, focusing on differences from displacementmeasurement device 100 according to Embodiment 1.

2-1. Configuration of Displacement Measurement Device 400

FIG. 10 is a block diagram illustrating the configuration ofdisplacement measurement device 400 according to Embodiment 2.

As illustrated in FIG. 10, displacement measurement device 400 isconfigured by changing displacement measurement device 100 according toEmbodiment 1 as follows: first machine learning model 10 has beenchanged to first machine learning model 410; second obtainer 60 has beenchanged to second obtainer 460; and first trainer 70 has been changed tofirst trainer 470.

First machine learning model 410 is a machine learning model trained togenerate at least one image which contains subject 300 and has noise,from one image which contains subject 300 and has noise, and isconfigured including a second machine learning model and a third machinelearning model.

FIG. 11 is a schematic diagram illustrating an example of theconfiguration of first machine learning model 410.

As illustrated in FIG. 11, first machine learning model 410 isconfigured including second machine learning model 411 and third machinelearning model 412.

Second machine learning model 411 is a machine learning model trained togenerate one image which contains subject 300 and has had noise removed,from one image which contains subject 300 and has noise. Second machinelearning model 411 is trained by first trainer 470. The training ofsecond machine learning model 411 by first trainer 470 will be describedlater.

Third machine learning model 412 is a machine learning model trained togenerate at least one image which contains subject 300 and has noise,from one image which contains subject 300 and has had noise removed.Third machine learning model 412 is trained by first trainer 470. Thetraining of third machine learning model 412 by first trainer 470 willbe described later.

After training, third machine learning model 412 is input with the oneimage which contains subject 300 and has had noise removed, generated bysecond machine learning model 411.

Second obtainer 460 obtains, from image capturing device 200, N imageswhich contain subject 300 and have noise, and one reference image whichcontains subject 300 and has had noise removed (i.e., which has nonoise). The N images and the one reference image obtained by secondobtainer 460 are used in the training of first machine learning model410 (described later). The images used in the training of first machinelearning model 410 (described later) are, for example, images capturedwhen there is no displacement in subject 300. Accordingly, secondobtainer 460 obtains N images and the one reference image captured whenthere is no displacement in subject 300.

When, for example, it is difficult for image capturing device 200 tocapture one image which contains subject 300 and has had noise removed,second obtainer 460 may obtain the one reference image which containssubject 300 and has had noise removed by obtaining an arithmetic mean ofN images, obtained from image capturing device 200, which containsubject 300 and have noise. Additionally, second obtainer 460 may obtainthe one reference image which contains subject 300 and has had noiseremoved by, for example, transforming one image containing subject 300,captured by image capturing device 200 in a state where image capturingdevice 200 has been moved close to subject 300 so that no noise ispresent, into an image projected at the same angle of view as the Nimages.

First trainer 470 trains first machine learning model 410 with each ofthe N images obtained by second obtainer 460 to generate at least oneimage which contains subject 300 and has noise from one image whichcontains subject 300 and has noise, by using each of the N images as aninput and using at least one of N−1 other images as a correct answer. Tobe more specific, first trainer 470 trains second machine learning model411 with each of the N images obtained by second obtainer 460 togenerate one image which contains subject 300 and has had noise removedfrom one image which contains subject 300 and has noise, by using eachof the N images as an input and using the one reference image obtainedby second obtainer 460 as a correct answer. Then, first trainer 470trains first machine learning model 410 by training third machinelearning model 412 to generate at least one image which contains subject300 and has noise from one image which contains subject 300 and has hadnoise removed, using the one reference image obtained by second obtainer460 as an input and using at least one of the other N−1 images obtainedby second obtainer 460 as a correct answer.

FIG. 12 is a schematic diagram illustrating an example of first machinelearning model 410 being trained by first trainer 470.

A specific example of the training of first machine learning model 410by first trainer 470 will be described next with reference to FIG. 12.In this example, image capturing device 200 captures a moving image ofsubject 300 when there is no displacement in subject 300, i.e., when aload acting on subject 300 is not changing, and second obtainer 460obtains N frames of the moving image captured by image capturing device200 as the N images which contain subject 300 and have noise. Then,first trainer 470 obtains the one reference image which contains subject300 and has had noise removed by finding an arithmetic mean of the Nimages which contain subject 300 and have noise.

As illustrated in FIG. 12, first trainer 470 sets, for each of the Nimages and the one reference image, at least one measurement point in apixel region where subject 300 appears. In the example illustrated inFIG. 12, first trainer 470 sets measurement point 301 and measurementpoint 302 in each of the N images and the one reference image, in thesame manner as illustrated in FIG. 4.

Then, first trainer 470 trains first machine learning model 10 with eachof the N images to generate at least one image which contains subject300 and has noise from one image which contains subject 300 and hasnoise, by using each of the N images as an input and using at least oneof the other N−1 images as a correct answer, and by performing thetraining on pixels included in each of the measurement points which havebeen set (measurement point 301 and measurement point 302).

The following will describe first trainer 470 as training first machinelearning model 410 to generate M images which contain subject 300 andhave noise.

To be more specific, first trainer 470 trains second machine learningmodel 411 with each of the N images to generate one image which containssubject 300 and has had noise removed from one image which containssubject 300 and has noise, by using each of the N images as an input andusing the one reference image obtained by second obtainer 460 as acorrect answer, and by performing the training on pixels included ineach of the measurement points which have been set (measurement point301 and measurement point 302). Then, first trainer 470 trains thirdmachine learning model 412 to generate M images which contain subject300 and have noise from the one image which contains subject 300 and hashad noise removed, by using the one reference image obtained by secondobtainer 460 as an input and the N images obtained by second obtainer460 as correct answers, and by performing the training on pixelsincluded in each of the measurement points which have been set(measurement point 301 and measurement point 302).

2-2. Operations of Displacement Measurement Device 400

Operations performed by displacement measurement device 400 configuredas described above will be described next.

Displacement measurement device 400 performs second learning processingin addition to the first displacement measurement processing performedby displacement measurement device 100 according to Embodiment 1. Thesecond learning processing performed by displacement measurement device400 will be described hereinafter.

The second learning processing is processing for training first machinelearning model 410 to generate at least one image which contains subject300 and has noise, from one image which contains subject 300 and hasnoise.

FIG. 13 is a flowchart illustrating the second learning processingperformed by displacement measurement device 400.

The second learning processing is started when a user using displacementmeasurement device 100 has performed an operation in displacementmeasurement device 100 for starting the second learning processing,when, for example, image capturing device 200 has captured a movingimage of subject 300 in which there is no displacement in subject 300and there is noise in each of the frames constituting the capturedmoving image.

When the second learning processing is started, second obtainer 460obtains, from image capturing device 200, N frames of the framesconstituting the moving image captured of subject 300, as the N imageswhich contains subject 300 and have noise (step S300).

Once the N images which contain subject 300 and have noise have beenobtained, second obtainer 460 obtains the one reference image whichcontains subject 300 and has had noise removed by finding an arithmeticmean of the N images which contain subject 300 and have noise (stepS310).

Once the one reference image containing subject 300 and having noiseremoved has been obtained, second obtainer 460 selects one image havingnoise from among the N unselected images having noise (step S320). Here,“unselected image” refers to an image, in a loop from the process ofstep S320 to the determination of “yes” in step S350 (described later),which has not been selected in a past instance of the process of stepS320.

Once one image having noise has been selected, first trainer 470generates one instance of first training data which takes the oneselected image having noise as an input and the one reference image as acorrect answer (step S330). Then, first trainer 470 generates oneinstance of second training data which takes the one reference image asan input and the one selected image having noise as a correct answer(step S340).

Once the one instance of first training data and the one instance ofsecond training data have been generated, first trainer 470 findswhether or not there is an unselected image among the N images havingnoise (step S350).

If, in the process of step S350, there is an unselected image (stepS350: yes), displacement measurement device 400 moves the sequence tostep S320 again.

If, in the process of step S350, there is no unselected image (stepS350: no), first trainer 470 uses N instances of the first trainingdata, obtained by repeating the loop formed by the process of step S320to the determination of “yes” in step S350 N times, to train secondmachine learning model 411 to generate one image which contains subject300 and has had noise removed from one image which contains subject 300and has noise (step S360). Then, first trainer 470 uses N instances ofthe second training data, obtained by repeating the loop formed by theprocess of step S320 to the determination of “yes” in step S350 N times,to train third machine learning model 412 to generate M images whichcontain subject 300 and have noise from the one image which containssubject 300 and has had noise removed (step S370).

Once the process of step S370 ends, displacement measurement device 400ends the second learning processing.

2-3. Effects

As described above, like displacement measurement device 100 accordingto Embodiment 1, displacement measurement device 400 generates the Mtemplate images which have noise from the first image which has noise,and generates the M target images which have noise from the second imagewhich has noise. Then, displacement measurement device 400 calculatesthe M hypothetical displacements from the M template images having noiseand the M target images having noise, and calculates the displacement ofsubject 300 by performing statistical processing on the calculated Mhypothetical displacements. Accordingly, like displacement measurementdevice 100 according to Embodiment 1, displacement measurement device400 can calculate the displacement more accurately than a past type ofdisplacement measurement device which calculates the displacement ofsubject 300 directly from a first image having noise and a second imagehaving noise, without performing statistical processing.

Embodiment 3

A displacement measurement device according to Embodiment 3, configuredby changing part of the configuration of displacement measurement device400 according to Embodiment 2, will be described next.

The displacement measurement device according to Embodiment 3 will bedescribed hereinafter, focusing on differences from displacementmeasurement device 400 according to Embodiment 2.

3-1. Configuration of Displacement Measurement Device 500

FIG. 14 is a block diagram illustrating the configuration ofdisplacement measurement device 500 according to Embodiment 3.

As illustrated in FIG. 14, displacement measurement device 500 isconfigured by changing displacement measurement device 400 according toEmbodiment 2 as follows: first machine learning model 410 has beenchanged to first machine learning model 510; first generator 30 has beenchanged to first generator 530; hypothetical displacement calculator 40has been changed to hypothetical displacement calculator 540; and firsttrainer 470 has been changed to second trainer 570.

First machine learning model 510 is a machine learning model trained togenerate at least one image which contains subject 300 and has had noiseremoved, from one image which contains subject 300 and has noise. Firstmachine learning model 510 is trained by second trainer 570.

Second trainer 570 trains first machine learning model 510 with each ofthe N images obtained by second obtainer 460 to generate at least oneimage which contains subject 300 and has had noise removed from oneimage which contains subject 300 and has noise, by using each of the Nimages as an input and using the one reference image obtained by secondobtainer 460 as a correct answer.

A specific example of the training of first machine learning model 510by second trainer 570 will be described with further reference to FIG.12, by taking FIG. 12 as a schematic diagram illustrating an example ofthe training of first machine learning model 510 by second trainer 570.In this example, image capturing device 200 captures a moving image ofsubject 300 when there is no displacement in subject 300, i.e., when aload acting on subject 300 is not changing, and second obtainer 460obtains N frames of the moving image captured by image capturing device200 as the N images which contain subject 300 and have noise. Then,second obtainer 460 obtains the one reference image which containssubject 300 and has had noise removed by finding an arithmetic mean ofthe N images which contain subject 300 and have noise.

As illustrated in FIG. 12, second trainer 570 sets, for each of the Nimages and the one reference image, at least one measurement point in apixel region where subject 300 appears. In the example illustrated inFIG. 12, second trainer 570 sets measurement point 301 and measurementpoint 302 in each of the N images and the one reference image, in thesame manner as illustrated in FIG. 4.

Then, second trainer 570 trains first machine learning model 510 witheach of the N images to generate at least one image which containssubject 300 and has had noise removed from one image which containssubject 300 and has noise, by using each of the N images as an input andusing the one reference image obtained by second obtainer 460 as acorrect answer, and by performing the training on pixels included ineach of the measurement points which have been set (measurement point301 and measurement point 302).

The following will describe second trainer 570 as training first machinelearning model 510 to generate M images which contain subject 300 andhave had noise removed. However, as another example, second trainer 570may train first machine learning model 510 to generate one image whichcontains subject 300 and has had noise removed from one image whichcontains subject 300 and has noise, and first machine learning model 510may generate M images which contain subject 300 and have had noiseremoved from one image which contains subject 300 and has noise bychanging a parameter of first machine learning model 510 M times whengenerating the image. As still another example, second trainer 570 maytrain first machine learning model 510 to generate one image whichcontains subject 300 and has had noise removed from one image whichcontains subject 300 and has noise, and first machine learning model 510may generate M images which contain subject 300 and have had noiseremoved from one image which contains subject 300 and has noise by usinga Generative Adversarial Network (GAN) when generating the image.

Returning to FIG. 14, the descriptions of displacement measurementdevice 500 will be continued.

Using first machine learning model 510, first generator 530 generates Mtemplate images which contain subject 300 and have had noise removedfrom the first image obtained by first obtainer 20, and generates Mtarget images which contain subject 300 and have had noise removed fromthe second image obtained by first obtainer 20.

First generator 530 has a similar function as first generator 30according to Embodiment 1, but with the following changes: “M templateimages having noise” has been replaced with “M template images havinghad noise removed”; and “M target images having noise” has been replacedwith “M target images having had noise removed”. Further detaileddescriptions of first generator 530 will therefore be considered to havealready been given, and will be omitted.

Hypothetical displacement calculator 540 calculates M hypotheticaldisplacements of subject 300 from the M template images which have hadnoise removed and the M target images which have had noise removed,generated by first generator 530.

Hypothetical displacement calculator 540 has a similar function ashypothetical displacement calculator 40 according to Embodiment 1, butwith the following changes: “M template images having noise” has beenreplaced with “M template images having had noise removed”, and “Mtarget images having noise” has been replaced with “M target imageshaving had noise removed”. Further detailed descriptions of hypotheticaldisplacement calculator 540 will therefore be considered to have alreadybeen given, and will be omitted.

3-2. Operations of Displacement Measurement Device 500

Operations performed by displacement measurement device 500 configuredas described above will be described next.

Displacement measurement device 500 performs third learning processingand second displacement measurement processing. The third learningprocessing and the second displacement measurement processing performedby displacement measurement device 500 will be described in orderhereinafter.

The third learning processing is processing in which the second learningprocessing according to Embodiment 2 has been partially changed, and isprocessing for training first machine learning model 410 to generate atleast one image which contains subject 300 and has had noise removed,from one image which contains subject 300 and has noise.

FIG. 15 is a flowchart illustrating the third learning processingperformed by displacement measurement device 500.

As illustrated in FIG. 15, compared to the second learning processingaccording to Embodiment 2, the third learning processing is processingin which the processes of steps S340 and S370 have been removed, theprocess of step S330 has been changed to the process of step S430, andthe process of step S360 has been changed to the process of step S460.Accordingly, the following descriptions will focus on the processes ofsteps S430 and S460.

In the process of step S320, once one image having noise has beenselected, second trainer 570 generates one instance of training datawhich takes the one selected image having noise as an input and onereference image as a correct answer (step S430). Displacementmeasurement device 500 then moves the processing to step S350.

If, in the process of step S350, there is no unselected image (stepS350: no), second trainer 570 uses N instances of the training data,obtained by repeating the loop formed by the process of step S320 to thedetermination of “yes” in step S350 N times, to train first machinelearning model 510 to generate one image which contains subject 300 andhas had noise removed from one image which contains subject 300 and hasnoise (step S560).

Once the process of step S370 ends, displacement measurement device 500ends the third learning processing.

The second displacement measurement processing is processing similar tothe first displacement measurement processing according to Embodiment 1,but with the following changes: “displacement measurement device 100”has been replaced with “displacement measurement device 500”; “firstmachine learning model 10” has been replaced with “first machinelearning model 510”; “first generator 30” has been replaced with “firstgenerator 530”; “hypothetical displacement calculator 40” has beenreplaced with “hypothetical displacement calculator 540”; “M templateimages having noise” has been replaced with “M template images havinghad noise removed”; and “M target images having noise” has been replacedwith “M target images having had noise removed”. Further detaileddescriptions of the second displacement measurement processing willtherefore be considered to have already been given, and will be omitted.

3-3. Effects

As described above, displacement measurement device 500 generates the Mtemplate images which have had noise removed from the first image whichhas noise, and generates the M target images which have had noiseremoved from the second image which has noise. Then, displacementmeasurement device 500 calculates the M hypothetical displacements fromthe M template images having had noise removed and the M target imageshaving had noise removed, and calculates the displacement of subject 300by performing statistical processing on the calculated M hypotheticaldisplacements. Accordingly, displacement measurement device 500 cancalculate the displacement more accurately than a past type ofdisplacement measurement device which calculates the displacement ofsubject 300 directly from a first image having noise and a second imagehaving noise, without generating images from which noise has beenremoved.

Embodiment 4

A displacement measurement device according to Embodiment 4, configuredby changing part of the configuration of displacement measurement device100 according to Embodiment 1, will be described next.

The displacement measurement device according to Embodiment 4 will bedescribed hereinafter, focusing on differences from displacementmeasurement device 100 according to Embodiment 1.

4-1. Configuration of Displacement Measurement Device 600

FIG. 16 is a block diagram illustrating the configuration ofdisplacement measurement device 600 according to Embodiment 4.

As illustrated in FIG. 16, displacement measurement device 600 isconfigured by changing displacement measurement device 100 according toEmbodiment 1 as follows: first generator 30 has been changed to firstgenerator 630.

Like first generator 30 according to Embodiment 1, using first machinelearning model 10, first generator 630 generates M template images whichcontain subject 300 and have noise from the first image obtained byfirst obtainer 20, and generates M target images which contain subject300 and have noise from the second image obtained by first obtainer 20.The methods through which first generator 630 generates the M templateimages and the M target images are different from those of firstgenerator 30 according to Embodiment 1.

FIG. 17 is a block diagram illustrating the configuration of firstgenerator 630.

As illustrated in FIG. 17, first generator 630 is configured includingapproximate displacement obtainer 631, second generator 632, and thirdgenerator 633.

Approximate displacement obtainer 631 obtains an approximatedisplacement of subject 300. Here, the “approximate displacement” ofsubject 300 is a displacement of subject 300 which is calculated inadvance and is not necessarily highly-accurate. For example, theapproximate displacement of subject 300 may be a displacement calculatedby a past type of displacement measurement device which calculates thedisplacement of subject 300 directly from a first image having noise anda second image having noise, without generating images from which noisehas been removed.

Second generator 632 generates a first pixel-relocated image byrelocating at least one pixel of the first image obtained by firstobtainer 20 an amount based on the approximate displacement obtained byapproximate displacement obtainer 631, and generates a secondpixel-relocated image by relocating at least one pixel of the secondimage obtained by first obtainer 20 an amount based on the approximatedisplacement obtained by approximate displacement obtainer 631.

Using first machine learning model 10, third generator 633 generates Mtemplate images which contain subject 300 and have noise from the firstpixel-relocated image generated by second generator 632, and generates Mtarget images which contain subject 300 and have noise from the secondpixel-relocated image generated by second generator 632.

4-2. Effects

When the displacement of subject 300 is relatively high, skew in pixelsincluded in the measurement points may exceed the ranges of themeasurement points.

In response to this, even if skew in pixels included in the measurementpoints exceeds the ranges of the measurement points, displacementmeasurement device 600 can use the approximate displacement to generatethe first pixel-relocated image, in which the skew in pixels included inthe measurement points does not exceed the ranges of the measurementpoints, from the first image, and to generate the second pixel-relocatedimage, in which the skew in pixels included in the measurement pointsdoes not exceed the ranges of the measurement points, from the secondimage. Then, using first machine learning model 10, displacementmeasurement device 600 generates the M template images from the firstpixel-relocated image and the M target images from the secondpixel-relocated image. In this manner, even if the displacement ofsubject 300 is relatively high, displacement measurement device 600 canaccurately calculate the displacement of subject 300.

Note that the displacement of subject 300 calculated by displacementmeasurement device 600 is a difference between the approximatedisplacement obtained by approximate displacement obtainer 631 and anactual displacement of subject 300. Thus to calculate the actualdisplacement of subject 300, it is necessary to add the approximatedisplacement obtained by approximate displacement obtainer 631 to thedisplacement of subject 300 calculated by displacement measurementdevice 600.

Embodiment 5

A displacement measurement device according to Embodiment 5, configuredby changing part of the configuration of displacement measurement device100 according to Embodiment 1, will be described next.

The displacement measurement device according to Embodiment 5 is adisplacement measurement device which aims to suppress a drop inaccuracy, caused by shot noise in a captured image, when measuring thedisplacement of a subject.

The displacement measurement device according to Embodiment 5 will bedescribed hereinafter, focusing on differences from displacementmeasurement device 100 according to Embodiment 1.

5-1. Configuration of Displacement Measurement Device 700

FIG. 18 is a block diagram illustrating the configuration ofdisplacement measurement device 700 according to Embodiment 5.

As illustrated in FIG. 18, displacement measurement device 700 isconfigured by changing displacement measurement device 100 according toEmbodiment 1 as follows: first machine learning model 10, secondobtainer 60, and first trainer 70 have been removed; first obtainer 20has been changed to obtainer 720; and first generator 30 has beenchanged to generator 730.

Obtainer 720 obtains, from image capturing device 200, a first imagewhich contains subject 300 and a second image which contains subject300. The first image and the second image obtained by obtainer 720 areused in the calculation of the displacement of subject 300. As such,obtainer 720 obtains the first image and the second image which havebeen captured during a period in which there is displacement in subject300 and/or a period in which there is no displacement.

Generator 730 generates M template images which contain subject 300 andhave noise from the first image obtained by obtainer 720, and generatesM target images which contain subject 300 and have noise from the secondimage obtained by obtainer 720. More specifically, generator 730generates the M template images by adding artificial noise to the firstimage, and generates the M target images by adding artificial noise tothe second image. Generator 730 sets, for each of the first image andthe second image, at least one measurement point in a pixel region wheresubject 300 appears. Each of the at least one measurement points is alocal region constituted by a plurality of pixels adjacent to eachother, and is a pixel region in a position that is the same in both thefirst image and the second image. Generator 730 adds the artificialnoise at the pixels included in the measurement points which have beenset. The artificial noise is random noise generated at random.

Generally speaking, shot noise is noise produced by statisticalfluctuations in the number of photons incident on a pixel per unit oftime, and the square of variance in the shot noise is proportional tothe number of photons incident on the pixel. Accordingly, the S/N ratioof shot noise in a pixel value read out from a pixel worsens as thenumber of photons incident on the pixel decreases, i.e., the lower thepixel value read out from that pixel is.

FIGS. 19A and 19B are schematic diagrams illustrating an example of aprobability density that a given pixel value will take on when lightproducing a given average pixel value is incident. In FIGS. 19A and 19B,the horizontal axis represents a pixel value read out from the pixel,and the vertical axis represents the probability density at which thatpixel value will occur. Assuming the pixel value read out from the pixelhas been quantized to 8 bits, FIG. 19A is a diagram corresponding to acase where the quantized pixel value read out from a pixel on whichlight having a peak probability density is incident is 1 (also called a“first case” hereinafter), and FIG. 19B is a diagram corresponding to acase where the quantized pixel value read out from a pixel on whichlight having a peak probability density is incident is 127 (also calleda “second case” hereinafter).

Here, for example, in the first case, the probability that the quantizedpixel value read out from the pixel will be 1 is 99.998%, due tostatistical fluctuations in the number of photons incident on the pixelper unit of time; and in the second case, the probability that thequantized pixel value read out from the pixel will be 127 is 10%, due tostatistical fluctuations in the number of photons incident on the pixelper unit of time.

As illustrated in FIG. 19A, in the first case, where the pixel valueread out from the pixel is comparatively low, fluctuation caused by shotnoise in the quantized pixel value which has been read out iscomparatively low as well. In other words, in the first case, thequantized pixel value which has been read out will be 1 with acomparatively high probability, namely 99.998%. However, as illustratedin FIG. 19B, in the second case, where the pixel value read out from thepixel is comparatively high, fluctuation caused by shot noise in thequantized pixel value which has been read out is comparatively high aswell. In other words, in the second case, the quantized pixel valuewhich has been read out will only be 127 with a comparatively lowprobability of 10%. In this manner, fluctuation in the quantized pixelvalue caused by shot noise becomes comparatively low when the number ofphotons of the light incident on the pixel is comparatively low, i.e.,when the pixel value is comparatively low, and becomes comparativelyhigh when the number of photons in the light incident on the pixel iscomparatively high, i.e., when the pixel value is comparatively high.

In a measurement point, when the pixel values of the pixels included inthat measurement point are comparatively high, the quantized pixelvalues of a comparatively large number of the pixels included in thatmeasurement point will fluctuate due to shot noise. Accordingly, takingthe measurement point as a whole, the fluctuation in the quantized pixelvalues of the pixels, which is caused by the shot noise, is averaged. Onthe other hand, in a measurement point, when the pixel values of thepixels included in that measurement point are comparatively low, thequantized pixel values of a comparatively small number of the pixelsincluded in that measurement point will fluctuate due to shot noise.Accordingly, taking the measurement point as a whole, the fluctuation inthe quantized pixel values of the pixels, which is caused by the shotnoise, is not averaged. Additionally, as described earlier, the pixelvalue of each pixel is relatively low from the outset, and thus the S/Nratio of the shot noise is comparatively poor. Due to these factors,when calculating the displacement of subject 300 based on the pixelvalues of the pixels included in a measurement point, fluctuations inthe pixel values caused by shot noise will have a comparatively highnegative effect on the accuracy of the calculated displacement when thepixel value of each pixel included in the measurement point iscomparatively low.

Accordingly, to suppress a drop in the accuracy of the calculateddisplacement, generator 730 adds artificial noise to pixels inaccordance with pixel values of pixels to which the artificial noise isto be added, so that as the pixel value decreases, a greater amount ofartificial noise is added.

5-2. Operations of Displacement Measurement Device 700

Operations performed by displacement measurement device 700 configuredas described above will be described next.

Displacement measurement device 700 performs third displacementmeasurement processing. The third displacement measurement processing isprocessing in which the first displacement measurement processingaccording to Embodiment 1 has been partially changed, and is processingfor calculating the displacement of subject 300 from the first imagewhich contains subject 300 and the second image which contains subject300.

FIG. 20 is a flowchart illustrating the third displacement measurementprocessing performed by displacement measurement device 700.

As illustrated in FIG. 20, compared to the first displacementmeasurement processing according to Embodiment 1, the third displacementmeasurement processing is processing in which the process of step S200has been changed to the process of step S600, and the process of stepS210 has been changed to the process of step S610. Accordingly, thefollowing descriptions will focus on the processes of steps S600 andS610.

The third displacement measurement processing is started when a userusing displacement measurement device 700 has performed an operation indisplacement measurement device 700 for starting the third displacementmeasurement processing, when, for example, image capturing device 200has captured a moving image of subject 300 during a period in whichthere is displacement in subject 300 and/or a period in which there isno displacement.

When the third displacement measurement processing is started, obtainer720 obtains, from image capturing device 200, one frame among the framesconstituting the moving image captured of subject 300 as the first imagewhich contains subject 300, and obtains another one frame as the secondimage which contains subject 300 (step S600).

Once the first image and the second image have been obtained, generator730 generates the M template images from the first image by addingartificial noise to the first image, and generates the M target imagesfrom the second image by adding artificial noise to the second image(step 610).

Once the process of step S610 ends, displacement measurement device 700moves to the process of step S220. Then, once the process of step S230ends, displacement measurement device 700 ends the third displacementmeasurement processing.

5-3. Effects

As described above, displacement measurement device 700 generates the Mtemplate images which have artificial noise from the first image, andgenerates the M target images which have artificial noise from thesecond image. Then, displacement measurement device 700 calculates the Mhypothetical displacements from the M template images having artificialnoise and the M target images having artificial noise, and calculatesthe displacement of subject 300 by performing statistical processing onthe calculated M hypothetical displacements. Accordingly, displacementmeasurement device 700 can suppress a drop in accuracy of thecalculation of the displacement of subject 300, caused by shot noise,more than a past type of displacement measurement device whichcalculates the displacement of subject 300 directly from a first imageand a second image, without performing statistical processing.Accordingly, displacement measurement device 700 can calculatedisplacement more accurately than the aforementioned past displacementmeasurement device.

Other Embodiments

Although one or more aspects of a displacement measurement deviceaccording to the present disclosure have been described thus far on thebasis of Embodiments 1 to 5, the present disclosure is not intended tobe limited to these embodiments. Variations on the present embodimentconceived by one skilled in the art, embodiments implemented bycombining constituent elements from different other embodiments, and thelike may be included in the scope of one or more aspects of the presentdisclosure as well, as long as they do not depart from the essentialspirit of the present disclosure.

(1) Embodiment 1 describes a configuration in which displacementmeasurement device 100 includes, for example, second obtainer 60 thatobtains N images which contain subject 300 and which have noise, andfirst trainer 70 that trains first machine learning model 10 using the Nimages obtained by second obtainer 60. However, as another example,displacement measurement device 100 may have a configuration in whichsecond obtainer 60 and first trainer are omitted, and first machinelearning model 10 is not trained. In this case, displacement measurementdevice 100 may use, for example, a trained first machine learning model10 which has been trained in advance by an external device or the like.

(2) Embodiment 1 describes the subject to be measured as a bridge, asone example. However, the subject to be measured need not be limited toa bridge. For example, the subject to be measured may be a structureaside from a bridge, such as a building or a steel tower, or may be aroad surface, a ball, an animal, or the like. When the subject to bemeasured is a subject which moves, such as a ball or an animal,“displacement of the subject” may be interpreted as meaning “movement ofthe subject”.

(3) In Embodiment 1, displacement measurement device 100 is described asbeing implemented in, for example, a computer device including aprocessor and memory, by the processor executing a program stored in thememory. However, as another example, displacement measurement device 100may be implemented in a computer system constituted by a plurality ofcomputer devices, each of which includes a processor and memory andwhich are communicably connected to each other, through distributedcomputing or cloud computing.

(4) Embodiment 1 describes a configuration in which, for example,displacement measurement device 100 does not include image capturingdevice 200. However, as another example, displacement measurement device100 may be configured including image capturing device 200. In thiscase, image capturing device 200 functions as an image capturer that isa part of displacement measurement device 100.

(5) Embodiment 5 describes a configuration in which displacementmeasurement device 700 generates the M template images by addingartificial noise to the first image, and generates the M target imagesby adding artificial noise to the second image. However, as anotherexample, displacement measurement device 700 may be configured togenerate the M template images by adding, to the first image, imagecapturing device-originating noise produced by an image capturing devicethat captured the first image, and to generate the M target images byadding, to the second image, image capturing device-originating noiseproduced by an image capturing device that captured the second image.Through this, displacement measurement device 700 can suppress a drop inthe accuracy of the calculation of the displacement of the subject,caused by image capturing device-originating noise present in thecaptured image. Additionally, for example, based on the pixel values ofthe pixels to which the image capturing device-originating noise is tobe added, displacement measurement device 700 may add the imagecapturing device-originating noise to those pixels so that as the pixelvalue decreases, a greater amount of noise originating from the imagecapturing device is added. The image capturing device-originating noisemay be, for example, dark current noise of the image capturing devicethat captured the image. Alternatively, the image capturingdevice-originating noise may be, for example, thermal noise of the imagecapturing device that captured the image.

(6) In Embodiment 1, some or all of the constituent elements included indisplacement measurement device 100 may be implemented by a singleintegrated circuit through system LSI (Large-Scale Integration).

“System LSI” refers to very-large-scale integration in which multipleconstituent elements are integrated on a single chip, and specifically,refers to a computer system configured including a microprocessor,read-only memory (ROM), random access memory (RAM), and the like. Acomputer program is stored in the ROM. The system LSI circuit realizesthe functions of the constituent elements by the microprocessoroperating in accordance with the computer program.

Note that although the term “system LSI” is used here, other names, suchas IC, LSI, super LSI, ultra LSI, and so on may be used, depending onthe level of integration. Furthermore, the manner in which the circuitintegration is achieved is not limited to LSI, and it is also possibleto use a dedicated circuit or a generic processor. It is also possibleto employ a Field Programmable Gate Array (FPGA) which is programmableafter the LSI circuit has been manufactured, or a reconfigurableprocessor in which the connections and settings of the circuit cellswithin the LSI circuit can be reconfigured.

Furthermore, if other technologies that improve upon or are derived fromsemiconductor technology enable integration technology to replace LSIcircuits, then naturally it is also possible to integrate the functionblocks using that technology. Biotechnology applications are one suchforeseeable example.

(7) Aspects of the present disclosure are not limited to thedisplacement measurement devices according to Embodiments 1 to 5, andmay be realized as a displacement measurement method which implementsthe characteristic constituent elements included in the displacementmeasurement device as steps. Additionally, aspects of the presentdisclosure may be realized as a computer program that causes a computerto execute the characteristic steps included in such a displacementmeasurement method. Furthermore, aspects of the present disclosure maybe realized as a computer-readable non-transitory recording medium inwhich such a computer program is recorded.

Although only some exemplary embodiments of the present disclosure havebeen described in detail above, those skilled in the art will readilyappreciate that many modifications are possible in the exemplaryembodiments without materially departing from the novel teachings andadvantages of the present disclosure. Accordingly, all suchmodifications are intended to be included within the scope of thepresent disclosure.

INDUSTRIAL APPLICABILITY

The present disclosure can be widely used in displacement measurementdevices that measure displacement of a subject to be measured.

What is claimed is:
 1. A displacement measurement device, comprising: anobtainer that obtains a first image which contains a subject to bemeasured and a second image which contains the subject; a generator thatgenerates M template images which contain the subject and which havenoise from the first image and generates M target images which containthe subject and which have noise from the second image, M being aninteger of 2 or higher; a hypothetical displacement calculator thatcalculates M hypothetical displacements of the subject from the Mtemplate images and the M target images; and a displacement calculatorthat calculates a displacement of the subject by performing statisticalprocessing on the M hypothetical displacements.
 2. The displacementmeasurement device according to claim 1, wherein the generator generatesthe M template images by adding noise to the first image, and generatesthe M target images by adding noise to the second image.
 3. Thedisplacement measurement device according to claim 1, wherein thegenerator generates the M template images by adding, to the first image,image capturing device-originating noise produced by an image capturingdevice that captured the first image, and generates the M target imagesby adding, to the second image, image capturing device-originating noiseproduced by an image capturing device that captured the second image. 4.The displacement measurement device according to claim 2, wherein basedon a pixel value of a pixel to which noise is to be added, the generatoradds the noise to the pixel so that as the pixel value decreases, agreater amount of noise is added.
 5. The displacement measurement deviceaccording to claim 4, wherein based on pixel values of pixels to whichthe image capturing device-originating noise is to be added, thegenerator adds the image capturing device-originating noise to thepixels so that as the pixel value decreases, a greater amount of noiseoriginating from the image capturing device is added.
 6. Thedisplacement measurement device according to claim 4, wherein the imagecapturing device-originating noise is dark current noise of the imagecapturing device that captured the image.
 7. The displacementmeasurement device according to claim 4, wherein the image capturingdevice-originating noise is thermal noise of the image capturing devicethat captured the image.
 8. A displacement measurement method,comprising: obtaining a first image which contains a subject to bemeasured and a second image which contains the subject; generating Mtemplate images which contain the subject and which have noise from thefirst image and generating M target images which contain the subject andwhich have noise from the second image, M being an integer of 2 orhigher; calculating M hypothetical displacements of the subject from theM template images and the M target images; and calculating adisplacement of the subject by performing statistical processing on theM hypothetical displacements.